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ABSTRACT 

Research conducted in various settings suggests that preparation and support for online instructors should 
be considered separately from comparable efforts for instructors in face-to-face environments. However, 
few studies provide empirical data that outline the ways in which preparation to teach online should 
differ, and only a handful link these practices to measurable outcomes that help define what is meant by 
effective. PBS TeacherLine’s model presents an opportunity to examine a comprehensive, well- 
established effort to prepare and support instructors of exclusively online courses. Results from a 
yearlong study illustrate PBS TeacherLine’s successes and explores their relationship with outcomes such 
as instructor retention, quality of online instruction, satisfaction with support received, and favorable 
learner course grades and re-enrollment rates. Findings reveal promising practices for preparing and 
supporting any online faculty, whether in k-12 or higher education, and suggest outcomes that may be 
appropriate for characterizing the effectiveness of these efforts. 
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I. INTRODUCTION 

As online learning has grown rapidly over the past decade, so has the need for effective ways of preparing 
and supporting high quality online instructors. Current research suggests the skills required to teach 
online are sufficiently different from face-to-face teaching to warrant separate guidelines and focused 
study [1, 2]. A logical next question is whether this difference warrants separate preparation and support 
(e.g. professional development, “PD”), and if so, what these efforts should entail. Various groups have 
articulated PD guidelines for online instructors, however very few studies provide empirical support for 
these guidelines, and only a handful link these practices to measurable outcomes that help define effective. 

The focus of this study was to better understand: 

What are successful strategies for implementing professional development to prepare and support 
instructors of online courses? 

What outcomes could meaningfully define PD for online instructors as effective? 
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This paper first reviews findings from current research, then describes PBS TeacherLine’s model for 
preparing and supporting online course instructors that have delivered their online courses to adult 
learners nationwide since its inception in 2000. A summary of methods and findings from a yearlong 
formative-summative research study on PBS TeacherLine’s PD model follows. Three cycles of data 
suggest key quality outcomes, captured from four key perspectives: PBS TeacherLine program 
administrators, coaches/mentors, PBS TeacherLine online course instructors, and course learners. 
Appendices provide the documents and procedures that PBS TeacherLine uses to support and evaluate 
online course instructors. 


II. HISTORICAL BACKGROUND 

Studies of face-to-face and online instructional environments agree the instructor plays a critical role in 
learner success [3], Yet while we know quite a lot about online learning, there is relatively little available 
research about online teaching. Examples of strategies that work best in certain teaching situations and 
measures for assessing online teaching abilities are sparse [4]. Many discussions about online teaching 
seem to agree that online instructors need PD that is designed specifically for their teaching environment 
[5] largely because online teaching requires different skills than teaching in face-to-face or hybrid 
environments [6]. Various groups such as the Southern Regional Education Board (SREB) and 
International Association for k-12 Online Learning (iNACOL) have responded by articulating standards 
for PD that support online instruction. Despite this progress, k-12 teachers and higher education faculty 
report a lack of adequate preparation to teach online and of support while they are teaching, with existing 
efforts focusing largely on course mechanics and technology details [7]. Opportunities to reflect on and 
enhance instructional skills as emphasized by professional standards are similarly limited [8]. 

While professional guidelines suggest principles that may result in well prepared, effective online 
instructors, little primary data exists to substantiate these principles in situ [9]. Available research is 
largely anecdotal and describes variations in goals, instructional strategies, media and technologies, time 
frames, incentives, and other variables of online PD programs [2], Further, there is a strong need for 
models that can support benchmarking across programs [4]. 

A. Recommendations for PD that serves online teachers 

Two agencies - the Southern Regional Education Board (SREB) and iNACOL - have set forth standards 
for PD that support online instructors spanning k-12 and higher education. SREB outlines the need for 
flexible, problem-based PD opportunities for teachers of online courses involving the use of multiple 
strategies [10]. iNACOL echoes SREB’s guidelines, underscoring the importance of mentoring and 
coaching for building social capital among instructors and for providing equitable access to PD across the 
career span [11]. iNACOL has also itemized indicators of high-quality PD that cross various online 
instructional roles including: technical skills, online learning pedagogy, state and national standards 
including content standards, serving students, curriculum, professionalism, and leadership [12], Broadly, 
SREB, iNACOL, and others converge on several foundational characteristics of PD for online instructors: 

• Offer a variety of PD experiences to meet the needs of an online instructor throughout his/her 
career. 

• Leverage social capital among instructors by partnering experienced and new teachers for 
extended mentoring and support and create forums where meaningful discussion can occur. 

• Develop systems to provide administrative monitoring and support for PD implementation 
including clearly stated expectations for instructor participation and performance in PD activities, 
as well as regular evaluation. 

• Ensure PD content: 

o Reviews current research and practices on effective online teaching 
o Illustrates models of effective online teaching, communications and strategies, and 
encourages instructors to role-play situations that apply these skills 
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o Includes a problem-based element with regular discussions about how to handle difficult 
situations with learners 

o Uses scenarios and case studies to differentiate instruction 

o Provides hands-on training and experimentation with the Learning Management System 
and other technology tools used to deliver and support instruction 

Some virtual high schools are designing professional development for their faculty that is differentiated to 
meet learners’ experience levels and learning needs, including content that aligns with the professional 
standards for online instruction mentioned above [13]. Few reports illustrate examples of PD to support 
online instructors in higher education, and of those that do, nearly all describe PD provided for online 
community college faculty. A common call is for PD that supports online instructors by incorporating 
adult learning theories, moves beyond course mechanics, recognizes the pedagogical differences of online 
teaching, and spans both k-12 and higher education [12]. 

B. PD Outcomes 

Research investigating the impact of PD on the quality of online instruction and on the quality of online 
learning, particularly among adult learners, is also limited. Reports include anecdotal accounts of 
effectiveness and/or some evidence of teachers’ positive attitudes toward PD activities, but much more is 
needed in order to relate specific aspects of PD implementation with outcomes of interest at various 
levels. Some outcomes have either been studied in the online environment or make sense to explore 
because of their significance to PD in general: 

• Online instructional skills (see below for PBS TeacherLine’s definition of these). 

• Understanding the roles and responsibilities of an online teacher [14, 15]. 

• Sense of efficacy as an online teacher [15]. 

• Interest in taking more PD [16]. 

• Evidence of professional learning through data-driven practice [17]. Formative assessment has 
been shown to be an effective mode of learning for instructors because the data collected are 
designed to support teaching modifications rather than for solely evaluating performance. While 
formative assessment can be short, medium or long-cycle, research indicates that short cycle 
formative assessment can have the biggest learning impact for inculcating a continuous 
improvement process [18]. 

• Instructor interactivity: Roblyer & Wiencke [15] proposed an early measure of online instructor 
skills in the online medium a rubric for assessing interactivity in an online class. 

• Reduced/low instructor turnover. 

• Learner course completion: One study concluded that strong faculty support translates into 
consistent learner course completion [19]. 

• Learner course grades: EDC [20] concluded that high-quality online PD for teachers could result 
in positive k-12 student outcomes such as higher student scores on state tests in mathematics and 
ELA. No comparable studies of the impact of PD on grades earned by adult learners were 
available at the time of this review. 

III. THE PBS TEACHERLINE MODEL 

Since the primary objective of this study was to provide empirical data on promising current practices, we 
needed to locate an online course provider that has been actively designing, implementing and refining 
efforts to prepare and support online instructors for some time. PBS TeacherLine has offered a catalog of 
more than 130 asynchronous online courses, taught by over 100 online course instructors for adult 
learners nationwide, since the program’s inception in 2000. From the beginning, PBS TeacherLine has 
been preparing and supporting online instructors with puipose-built PD refined regularly to address many 
of the professional standards set forth by SREB and iNACOL. 

PBS TeacherLine’s PD model presents an opportunity to examine a comprehensive, well-established 
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effort that has undergone regular refinements guided by learner and instructor feedback. Although the PD 
model was originally developed specifically for PBS TeacherLine’s online course instructors, the research 
basis, adult population of learners, and variety of background represented by PBS TeacherLine instructors 
suggest that key elements could be transferred to other settings including k-12 and higher education. A 
study of PBS TeacherLine’s efforts includes these additional advantages: 

• Focused study of the elements that work best in a well-defined, exclusively online environment. 

• Online instructors as learners who move beyond course mechanics and implementing the LMS, 
toward enhancing professional skills 

• Peer review and formative assessment, applying research-based practices for adult learning and 
continuous improvement. 

• Focused on high-quality online instruction without the parallel responsibility for course 
development. 

• Course content and design modeled on inquiry based adult learning [21], incorporating principles 
that have been tested in multiple learning environments with understood strengths and limitations, 
with concrete activities that can be measured and about which researchers can collect data. 

• Dedicated resources for supporting and developing online instructors. 

• Four six-week terms in one calendar year for rapid adjustments and feedback, as well as efficient 
study. 

• As an independent provider, PBS TeacherLine has the flexibility to select and implement best 
practices from three key sectors: k-12, higher education and industry. 

PBS TeacherLine strives to provide high quality online instruction by first selecting individuals who fit 
specific selection criteria (Figure 1), then providing initial training, ongoing support and regular feedback 
to assure continued quality. This study focuses on the primary PD elements of instructor preparation, 
support, and feedback. 


Select 


• Prior online learner 

• MS/MA 

• Relevant content 
expertise 

j 


Prepare 



A 

•OFTI 


• Mentoring 


• OFT II 


V 

V 


Support 

Feedback 

•FUN 

• 3D course reviews 

• Resource library 

• Learner 

• Faculty meetings 

• PBS TL program staff 


• Instructors 


V_ J 


Figure 1. Key components of PBS TeacherLine’s instructor PD model 


The key features of PBS TeacherLine’s model are featured in Table 1 and described below. Once hired 
by PBS TeacherLine’s Associate Director of Online Facilitation, new instructors begin their initial 
preparation with PBS TeacherLine by enrolling in two online courses— Online Facilitation I (OFTI) and 
Online Facilitation II (OFTII) —and receive mentoring support while they teach their first online course. 
With more online teaching experience, instructors move into an annual cycle of self-reflection and course 
review that is designed to promote inquiry-based thinking about their professional practice and provide a 
forum for constructive professional discussions about problem-based instructional issues that take place in 
a moderated professional learning community called the FUN (Facilitators’ United Network). 
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Strategies 

PBS TeacherLine PD component(s) 

OFT 1 

OFT 

11 

Mentoring 

FUN 

Faculty 

meetings 

3D course 
reviews 

Peer support 

X 

X 

X 

X 



Ongoing, embedded 
guidance 

X 

X 

X 

X 

X 

X 

Problem-based 

X 

X 

X 

X 

X 

X 

Incorporates primary data 

X 

X 

X 

X 


X 

Delivered online 

X 

X 

X 

X 

X 

X 


Table 1. PD guidelines addressed by PBS TeacherLine’s model 


Tables 2 and 3 below explain the various roles that will be discussed and analyzed throughout the paper. 


Role 

Description 

Course learner 

A K-12 educator who is seeking state/district recertification and/or 
professional development. 

Instructor 

A course instructor who is a master’s level or above with previous 
experience teaching and learning online and must take the PD courses 
provided by PBS TeacherLine. 

Mentor 

An experienced TeacherLine instructor who has taught more than 20 
courses. 


Table 2. PD Participants 


Measure 

Detail 

Response Rate 

Appendix 

1. 3D course 
review 



A 

2. 

Background 

survey 

• Demographic information 

• Prior experience with and 
perceptions of PD that instructors 
have received to date 

n=94/104 (90.4%) of PBS 
TeacherLine’s instructors 

B 






a. 

Reflection 

logs 

All instructors were encouraged to 
minimally complete one log during each 
term they taught. 

Level 1: 

• Guided self-critique of online 
teaching 

• Provided data for 3D course 
reviews 

• Instructors used a three-point* 
scale to rate their: 

• Course climate and community 
building 

• Instruction 


Active 

Tnstriiptnrs; 

Completed Log 
Lvl 

Completed Log 

Lvll 

Did not 
participate 



F 

09 

5 

9 

47 

- 

12 

W 

10 

6 

2 

14 

38 

0 

Sp 

10 

6 

4 

10 

51 

3 
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• Interaction and discussion 
facilitation 

• Course organization 

• Implementation of assessments 

• Used by instructors to set goals 
and self-assess instructional skills. 

Level II 

Above, plus: 

• Indicates completed Level 1; 
appropriate for a more 
experienced online course 
instructor 

• Guided data-driven critique of 
prior instructional successes and 
quantification of movement 
toward achieving goals 



b. Post¬ 
course 

learner 

survey 

• All learners were surveyed at the 
end of each course. 

• Included scales for rating 
instructor skills according to the 
same dimensions that instructors 
and PBS TeacherLine as part of 
the 3D course reviews. 

• Gauged learners’ intentions for 
completing more courses and/or 
recommending courses to 
colleagues 

Learners 

n=2120/4197 enrolled 
(50.1%) 

C 

c. Course 

review 

rubric 

• Completed by the Associate 
Director for Online Instruction, 
which includes scales for rating 
instructor skills that mirror other 
instruments in the 3D review 
process. 

• Includes area for specific 
comments 

• Conducted at least annually for 
each online course instructor 

One completed for each 
instructor during each course 
term (n=178 ) 

H 

3. Post¬ 
course 
telephone 
interviews 

• Explored TeacherLine’s PD 
model in greater detail 

• Feedback on the process of 
completing reflection logs was 
also collected during the 
interviews. 

• 9 facilitators interviewed in total; 

3 in each group stratified by 
having completed OFT I, OFT 1 
with mentoring, and OFT 11 

Instructors (n=9) and PBS 
TeacherLine program staff 
(n=4) 

F 
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• 15-20 minutes long 



4. Learner 
course grades 
and re¬ 
enrollment 
data 

• PBS TeacherLine assigns course 
grades of A, B, C, D and F. 

• Re-enrollment statistics for all 
learners and courses offered 
during the study 

100% of course learners; 
n=4197 

N/A 


*A 3-point scale was used during the Fall 2009, Winter 2010 and Spring 2010 terms. A 5-point scale was piloted during the 
summer of 2010 and is recommended for future use. 


Table 3. Measures and participant 

A. Preparing High Quality Online Course Instructors 

In PBS TeacherLine’s Online Facilitator Training(OFT) I: Mastering the Skills of Online Teaching, all 
instructors that are new to PBS TeacherLine engage with course content in an asynchronous environment 
that includes online discussions and collaboration with other PBS TeacherLine instructors-in-training. 
OFTTs course content includes a variety of fundamental concepts and practices such as: 

Philosophy and pedagogy of online learning 

Techniques for deepening online discussion 

Techniques for online, text-based communication 

Strategies to foster online community building and collaborative learning 

Strategies for delivering effective, formative online assessment 

Course facilitation and administration policies 

Strategies for evaluating Web-based resources 

In a follow-up professional development experience with OFT I plus Mentoring, new instructors 
undertake the leadership of their own PBS TeacherLine course with the guidance and support of an 
experienced instructor mentor. Mentors, while non-evaluative in their role, provide valuable feedback 
and affirmation for beginning instructors by acting as onlookers to instructor-learner communications, 
advisors for consultation regarding learner questions, partners in dialog about best practices, and resource 
sharers. PBS TeacherLine provides mentors with a checklist with structured guidance for the mentor- 
instructor relationship. 

Instructors next enroll in PBS TeacherLine’s second online course, OFTII: Adjusting (or Dusting) your 
Facilitator Hat. OFT II provides more experienced instructors with additional research-based 
information about online teaching and learning and is a medium through which they can practice their 
instructional skills. OFT II fosters data-driven practice through instructors’ analysis of their own previous 
course evaluations or relevant examples of online learning. 

B. Ongoing PD for supporting established online course instructors 

After OFT /, mentoring and OFT II are completed, instructors’ primary mode of professional 
development is the 3D course review process. Course reviews integrate principles of formative and 
summative evaluation to help structure and contextualize professional growth among online faculty. 3D 
reviews take three perspectives on a course’s instruction—instructor, PBS TeacherLine program staff and 
learners—and put them side by side for a specific account of particular strengths and areas of needed 
improvement (see Appendix A for an example). PBS TeacherLine’s 3D course reviews were designed 
primarily to promote self-reflection among instructors, to increase professional dialogue about concrete 
instructional scenarios and to better identify PD needs. Secondarily, they served as artifacts that could be 
used to explore learner trends, review patterns across courses, and convey information about instructional 
performance. A Learner Feedback Summary cover page (Appendix D) aggregated all learner data 
collected during a course term to contextualize feedback from learners, PBS TeacherLine program staff, 
and instructors pertaining to specific course(s). Then, instructors met with PBS TeacherLine program 
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staff to discuss the course reviews and define instructional goals for the next teaching cycle. 

Finally, embedded learning opportunities occurred through faculty meetings, a professional learning 
community of instructors called the FUN, and a resource library that support instructors’ ongoing 
information needs in real time. Quarterly faculty meetings conducted by PBS TeacherLine’s Associate 
Director of Online Facilitation are held via webinar and feature a Web 2.0 instructional demonstration 
(e.g. Voki) along with housekeeping agenda items. The FUN offers an online professional learning 
community that is available to all PBS TeacherLine course instructors, and exists almost entirely as a self- 
directed forum. The FUN provides supplemental just-in-time professional development by reinforcing 
good communication and providing access to static information that is always available. Also available is 
a library of documents that can be easily downloaded, printed, and/or referred to at any time through the 
course management system. 

IV. METHOD 

The yearlong case study began during the fall of 2009 and encompassed three consecutive PBS 
TeacherLine course terms—Fall 2009, Winter 2010, and Spring 2010. Data were collected from all of 
PBS TeacherLine’s instructors at the time, who were required to complete at least a background survey. 
Instructors also were required to participate in 3D course reviews for each term of the study during which 
they taught a course. Online surveys and telephone interviews with instructors and key PBS TeacherLine 
program staff provided data that contextualized quantitative data collected during the study such as 
learner grades, enrollment figures and course reviews. Efforts were made to secure feedback from every 
learner enrolled in a course during all terms of the study. 

A. Measures 

Balancing the opportunity to collect rich data from an entire online faculty was a need for instructors’ 
involvement in the research to be meaningful. The central methods for collecting data on each element of 
PBS TeacherLine’s model were a background survey completed by 92.9 percent (n=94) of the faculty in 
October 2009, 3D course reviews completed by an average of 85% of active instructors during a given 
term, and learner grades collected from all learners who completed courses during the study (Table 2.) 
All instructors completed the appropriate log, had their courses reviewed by PBS TeacherLine program 
staff, and secured learner feedback about their course during each of the terms they taught while the study 
was underway as part of the 3D course review process described above. 

• 3D Course Reviews (Appendix A)—resulted from following the guidelines described in 
Implementing 3D course Reviews (Appendix 1) for using three instruments to aggregate data into 
a comprehensive report. 

• A general background survey (Appendix B) deployed to all online course instructors at the start 
of the study gathered feedback about their preparation to teach online and the PD they had 
received to date. 

• Reflection logs (Appendix C)-Logs featured closed- and open-ended questions that guided 
instructors through a critique of their own teaching in five key skill areas during each course 
term: course climate and community building, instruction, interaction and discussion facilitation, 
course organization and implementation of assessments [22], Reflection logs graduated from 
Level 1 to a Level 11, indicating a completed Level 1 log and prior participation in the 3D course 
review process. Level 1 logs asked instructors to rate their own performance during a course, 
while Level 11 logs asked instructors to also review and comment on the findings revealed by a 
prior course review. Instructors used the logs to set goals and identify their strengths and 
weaknesses. 

PBS TeacherLine Course Review Rubric (Appendix D)—PBS TeacherLine program staff provided data 
on course instruction during the study through the course review rubric embedded in the 3D review 
process. PBS TeacherLine’s Associate Director of Online Facilitation rated every instructor’s course 
during each of the three study terms that (s)he taught a course using a rubric with the same six 


128 


Journal of Asynchronous Learning Networks, Volume 16: Issue 2 



Toward an Understanding of What Works in Professional Development for Online Instructors: 

The Case of PBS Teacherline 


instructional skill areas as those featured in the instructor self-reflection logs and learner surveys 
stemming from PD activities. Artifacts such as discussion board activity, landscape posts, and dialogue 
about learner assignments and assessment represented instructional and learning activities during the 
course. 

Post-Course Learner Survey (Appendix E) - Course learners completed an online survey about their 
perceptions of instructional quality and course organization during the course’s final days. The post¬ 
course learner survey included scales for rating instructor performance according to the same five 
dimensions that instructors and PBS TeacherLine used to assess instruction as part of the 3D course 
reviews. 

• Interviews (Appendix F) - Telephone interviews with instructors and key PBS TeacherLine 
program staff provided data that contextualized quantitative data collected during the study such 
as learner grades and enrollment figures. 

• Learner Data -Grades and re-enrollment rates were compiled for all learners in all courses offered 
during the study’s three terms. 

An evaluative program rubric (Appendix G) was also explored during the study to synthesize all of the 
data collected and facilitate a better understanding of faculty-wide strengths and needs. The program 
rubric awards points (0, 1, 2, 3) for the combination of factors PBS TeacherLine defines as priorities: 
instructor qualifications (11 points total) and performance (31 total points) using assessment data 
collected through the background surveys and 3D course reviews from PBS TeacherLine. The rubric can 
also be used to highlight those instructors that have shown a superior or distinguished record of online 
instruction. All rubric elements align directly with PBS TeacherLine’s clearly defined expectations for 
instructor qualifications, performance and participation in PD. 

B. Participants 

Over the course of the study, each of PBS TeacherLine’s approximately 110 online course instructors 
participated in the research, though their involvement varied during the study depending on whether and 
when they taught courses (Table 2). PBS TeacherLine instructors are recruited through local PBS stations 
and are located in various geographic locations across the United States. As of the fall 2009 course term, 
instructors were in various stages of PBS TeacherLine’s PD model: OFT I, (19 instructors; 18.8 percent 
of the entire faculty), mentoring (12; 11.9%), and OFT II (60; 59.4%). Completion of the background 
survey was mandatory for all instructors, and each of the 94 instructors who completed the survey was 
paid $50.00 cash. The first three instructors from each of the three groups to confirm interest were 
included in interviews for this study. Instructors that completed reflection logs were entered in a drawing 
to win one of five $50.00 gift cards for a major online retailer at the end of each course term. 

All course learners were asked to complete the post-course learner survey as part of the 3D course 
reviews. Response rates among course learners increased from 41.3 to 65.1 percent during the study, 
potentially corresponding with the value that instructors placed on the multiple data sources compiled by 
the 3D review process. As with instructors, learners who completed surveys were entered in a drawing to 
win one of ten online gift cards at the end of each course term. 

C. Analysis procedures 

Quantitative data from reflection logs, surveys and rubric ratings underwent descriptive analysis (mean, 
range, standard deviation) to identify and differentiate trends according to the study’s initial guiding 
questions. Learners’ final course grades from all three terms were correlated with facilitator quality to 
explore the relationship between effective ongoing instructor support and learner outcomes. Qualitative 
data collected through interviews and reflection logs were collated and coded using Grounded Theory 
[23], which allowed raw data to be easily compared and contrasted to reveal common participant 
experiences as well as those that are unique to individuals. Quantitative and qualitative data were 
compared to identify similarities and differences that inform the study’s final conclusions. 
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V. RESULTS 

PBS TeacherLine strives to ensure high quality instruction by clearly defining and modeling the skills that 
contribute to learner success in an online environment, and then by providing initial training, ongoing 
support and regular feedback to assure online course instructors demonstrate these skills. Positive 
outcomes suggest the program is successful in several key ways. 

A. PD for online instructors 

PBS TeacherLine’s commitment to providing high quality PD for its online instructors has contributed 
substantially to the program’s decade-long history of success. The PD model overall demonstrates key 
successes in each of its four phases: selection, preparation, support and feedback (Figure 1), which 
substantiate some of the current PD guidelines for online instructors. 

1. Select qualified instructors 

As with any program, PBS TeacherLine’s instructors must meet minimum qualifications. Before 
teaching their first course, all of PBS TeacherLine’s instructors must hold a master’s degree (16.3% hold 
CAS/EdS or PhD degrees), have prior experience as online learners, and also have subject-area teaching 
experience in at least one of the four areas in which PBS TeacherLine offers courses: Mathematics, 
Science, Reading/Language Arts, and Instructional Technology. Prior first-hand experience as an online 
learner may be the most critical qualification for instructors who proceed through a PD program such as 
PBS TeacherLine’s that emphasizes teachers as learners engaged in critical analysis of the online learning 
experience. Prior online learning exhibited a bi-modal distribution among PBS TeacherLine’s instructors 
during the study, with two-thirds having enrolled in six or fewer online courses and the remaining third 
having enrolled in 30 or more online courses as learners before the Fall 2009 course term. PBS 
TeacherLine also seeks instructors that are actively teaching in higher education, k-12 or both, though 
they do not necessarily have to be teaching online. Half had been teaching online for more than five years 
as of the fall 2009 term. Nearly all of the experienced instructors had been with PBS TeacherLine for 
their entire online teaching career. 

2. Define the PD 

While PBS TeacherLine’s program model of separating course development from delivery provided a 
natural starting point, PD designers further established a primary focus on instructional skills with a 
secondary focus on course development and mechanics. The emphasis on instructors and instructional 
skills may have presented an advantage since50 instructors (54.3% of PBS TeacherLine’s faculty) felt 
that no other provider offered opportunities to deepen their online instruction skills in the way that PBS 
TeacherLine’s PD has. 

For PBS TeacherLine, PD includes activities geared toward instructors who are in various stages of their 
online teaching careers. In developing its model, PBS TeacherLine asked questions such as: 

What does good online instruction look like? 

What roles do our instructors fill? 

How can we measure our instructors’ progress in these areas? 

What do instructors need to know? When? 

What do our instructors do well? In what areas would they benefit from additional professional 
development? 

3. Make PD relevant 

PBS TeacherLine addresses instructors’ needs for contextualized and relevant PD in two key ways: by 
encouraging critical analysis of real teaching and learning situations in all PD activities, and by 
connecting these analyses to a customized portfolio (3D course review) that demonstrates self-assessment 
and progress in instructional skills over time. In both ways, instructors are engaged in cycles of using 
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data to inform their instruction and creating documents that capture feedback for meaningful analysis. 
Clearly stated and demonstrated expectations for high quality instruction that originated with research- 
based practices are woven throughout the sequence of activities, with embedded opportunities to address 
targeted issues related to the LMS or course mechanics. 

4. Implement a variety of PD strategies 

A combination of sequential online courses, mentoring, professional learning communities, and a library 
of digital resources contributed to the success of PBS Teacherline’s PD model. In addition to online 
courses and the 3D review process, the FUN, faculty meetings, and resource library played important 
support supporting roles by providing just-in-time references and timely information. As one instructor 
wrote, 

It is a great source of comfort to have the support of other instructors 1 can interact with in the PD 
courses and on the FUN. This is especially true when a difficult situation arises. It is good to get 
advice from others who have been in the same situation, and who have gone through the same PD 
courses. The sample templates, responses, and documents that experienced instructors provide in 
the FUN are extremely helpful because they connect with and extend the concepts presented in the 
courses, reinforced through mentoring and measured in the [3D] course reviews. 

The FUN leveraged social capital among PBS TeacherLine’s online faculty to reinforce good online 
communication skills and provide access to digital references whenever they needed it. Although nearly 
four in ten instructors were lurkers who referred to the FUN to gain information and/or network more 
than they contributed to the site, nearly all (n=92) instructors valued the exchanges that took place there. 
Three in ten instructors felt the FUN was essential to their success as TeacherLine course instructors. 

5. Feature rigorous PD content 

PBS TeacherLine’s PD content reflects its goals and design by featuring: 

Research-tested strategies that highlight effective ways of teaching and learning online. 

PBS TeacherLine ensures that guidelines cited in PD offerings reflect current research and discourse on 
online teaching and learning. One instructor described how research-supported strategies were the most 
important factor in determining the value of PD for her, “I only take PD that illustrates how the strategies 
and techniques offered reflect key findings from research on how adult learners leam best in an online 
environment. PBS TeacherLine is the only provider I have found that does this for their online 
instructors.” Eighty-seven (92.6%) of PBS TeacherLine’s instructors said it is essential for PD to focus on 
research-tested instructional strategies that provide evidence of positive impacts on learner outcomes, and 
all 87 instructors said that PBS TeacherLine’s PD does this “very well.” 

Extensive modeling of high quality online instruction and guided practice to scaffold learning. 

PBS TeacherLine’s PD offers modeling and guided practice by positioning online teachers as learners in 
the online environment. OFT I and II have instructors model effective online teaching, communications 
and strategies, and role-play situations that apply these strategies. During the study, 89 instructors 
(94.7%) said that hands-on PD with extensive modeling and guided practice involving the use of new 
techniques is critical to their professional growth. 

Embedded learning opportunities with immediate applications to real world environments. 

Real world applications of PD content are as critical to online instructors as relevant course content is to 
learners. For example, one instructor said: 

[OFTI and OFTIT\ have provided the help needed to understand how TeacherLine’s PD differs 
from, and is superior to, online courses from other providers. The inclusion of practical 
applications is, for me, essential, and both courses include them. 

Instructors receive hands-on training and experimentation with the Learning Management System and 
other technology tools used to deliver and support instruction as embedded components of PD activities. 
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6. Adopt a data-driven culture 

PBS TeacherLine continually strives to support high-quality online instruction by regularly evaluating 
and updating their PD content and mode(s) of delivery. In online teaching and learning, we have always 
had data about learners such as enrollment and grades. We collect relatively less data on instructors and 
have only begun to do so recently. By contrast, PBS TeacherLine has collected data on learners, 
instructors and courses from the very beginning, and provided some learner feedback to instructors after 
courses end. The Associate Director of Online Facilitation has traditionally observed course artifacts 
such as discussion board activity and assessment feedback, and reported back to instructors along with 
limited learner feedback gathered at the end of the course. Recently, they have aligned data sources, 
measures, and procedures through the 3D course reviews to generate more meaningful information about 
online instruction to assist the program and also instructors. PBS TeacherLine’s data-driven culture 
features: 

• Leveraging the inherent technology in an online environment to provide accurate, timely and 
iterative feedback about instructional quality to instructors, prompts for responding to the 
feedback, and an asynchronous forum for sharing ideas can meaningfully support instruction. 

• Aggregating, contextualizing, and sharing data for easy analysis and discussion. 

• Defining and implementing a formal review process that connects back to research-tested 
teaching strategies in the online classroom. 

• Transparency, with data collected from many sources. 

• Flexibility to grow and change the PD model with the needs of the faculty it serves. 

The 3D review process arose from a need to accurately measure PD quality and instructional performance 
from multiple perspectives and aggregate data for easy analysis and discussion. While learners are still 
coming on board with their role in giving feedback (only about 50% of learners completed surveys at the 
end of their courses), nearly all instructors participated actively in the 3D reviews during the study, 
suggesting their early buy in. 

3D course reviews contributed positively to a culture of continuous improvement and provided a 
meaningful framework for ensuring instructional quality during the study (see tables below.) Instructors 
used their reviews in a variety of ways, most commonly to identify their strengths and weaknesses. They 
appreciated seeing primary data about their own instruction from learners and PBS TeacherLine’s 
administrators alongside critical thinking questions to prompt future instruction. Two in three instructors 
rated the course review process as ‘very useful’ and the remaining third considered their reviews to be 
‘somewhat useful’. Eighty-five percent of instructors felt the course review process was meaningful ‘as 
is’ without the need for updating or changing. 

B. PD outcomes 

PBS TeacherLine’s implementation of these PD guidelines has received positive feedback from 
instructors, and key skills emphasized by PD are evident among instructors as rated by PBS TeacherLine 
program staff, course learners, and themselves. 

1. PBS TeacherLine’s instructors consistently received high quality ratings from PBS 
TeacherLine program staff and their learners in the skill areas emphasized by PD 
content 

The program rubric and contributing data from 3D course reviews paint a picture of high quality online 
instruction across all five online teaching skills measured during the study (Table 4.) There was high 
overall agreement between learners’ (averaged by course) and PBS TeacherLine’s course ratings assigned 
by program staff, with instructors’ self-assessment scores consistently lower during the study. 
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Instructional skill 

Mean 

instructor self¬ 
rating 1 ’ 2 

Mean PBS 
TeacherLine 
rating 1 

Mean learner 
rating 1 

Course climate and community building 

2.7 

2.9 

2.9 

Instruction 

2.6 

2.9 

2.8 

Interaction and discussion facilitation 

2.6 

2.9 

2.8 

Course organization 

2.7 

2.8 

2.8 

Implementation of Assessments 

2.6 

3.0 

2.8 


1:3 = Excellent 2 = Satisfactory 1 = Needs improvement 
2: Level I and Level II log data combined 

Table 4. Aggregate ratings of instructional quality across three course terms 


The program rubric contextualizes these ratings with other key experiences that PBS TeacherLine 
emphasizes in its preparation and support model such as prior online learning and qualifications to teach 
online. Among 48 instructors with sufficient data during at least one course term, 33 (69/6%) displayed 
superior quality by achieving Master status, with another 15 (31.3%) achieving Distinguished status 
(Figure 2.) All input data sources (PBS TeacherLine, instructor, and learner) must be represented with 
feedback secured from at least 50% of an instructor’s learners in order to generate an accurate score. 



43 40 39 38 37 36 35 34 33 32 31 27 24 23 21 19 18 17 7 


Quality Score (Program Rubric) 


Range of possible scores: 0-48 

Figure 2. Histogram of instructor quality scores 

Learners most appreciate how their instructors make it easier to implement course concepts in their own 
environments as was modeled for instructors in their PD. As one learner said, 

Our instructor discussed how she uses technology as a teacher, creating an environment of 
modeling vs. “do what I say.” She walks the walk! [She] was very timely with grading projects 
and thorough on her very personalized comments. 

Nearly all (94.5%) learners said they could immediately apply what they learned during a TeacherLine 
course in their own professional practice, 98.2 of which had done so before the course was over. Learners 
rate instructors’ ability to facilitate the application of their learned knowledge very highly. 

Instructors also articulated areas in which they could improve their skills: landscape posts, presence on 
the discussion board, and constructive, valuable and specific feedback to learners. Learners’ ratings in 
three skill areas were slightly lower than those of PBS TeacherLine program staffs ratings: instruction, 
interaction and discussion facilitation, and course organization. 
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2. Instructors were satisfied with the preparation and ongoing support they have received 
from PBS TeacherLine. 

While instructor competencies cannot be solely attributed to PBS TeacherLine’s PD offerings, instructors 
unanimously agree PBS TeacherLine’s PD model has been effective for helping them define, practice and 
improve their online teaching skills. One instructor said, “PBS TeacherLine’s PD shows you exactly 
what you need to know to administer a course, whether you’re just getting started or you’ve been around 
a while like 1 have.” Nearly all of PBS TeacherLine’s instructors gave very high ratings to the content 
and format of OFT I and OFT II describing the required courses as “outstanding”, “dynamic”, and 
“relevant”. They particularly valued the online discussions embedded within the courses for modeling 
their own discussion board leadership in future courses. Instructors attributed their professional growth in 
online instructional skills to participating in PBS TeacherLine’s PD. Specifically, instructions said: 

• 1 think that the 3D course reviews are excellent. The evaluation feedback is the most valuable and 
has led me to improve. 1 love seeing the bigger picture too. 

• OFT 11 had the most positive impact for me. 1 learned so much about using the rubrics for 
assessment and how to create interventions. 

• 1 believe OFT II has been the most beneficial training to date because it opened up a world of 
growth for me. 

Both mentors and mentees appreciated the mutual affirmation and problem solving that takes place within 
the mentoring relationship. As one instructor mentee stated, 

My mentor affirmed that my instincts were correct, which I really needed. She also covered 
content that OFT I couldn’t by providing examples of correspondence with learners that I 
couldn’t find in other places. Sometimes there’s a situation that we just didn’t cover in OFT I and 
having my mentor’s support as I worked through it was invaluable to me. 

3. High quality PD contributes to a very low turnover rate among online course 
instructors. 

PBS TeacherLine’s loyal faculty has remained relatively stable over the past decade (Figure 3), with 
average instructor tenure of five years. One instructor summarized the feedback from 85 others who cited 
PBS TeacherLine’s high quality PD as a significant factor in wanting to stay, “the PD has confirmed that 
1 am working with professionals who know and care about what they are doing. All clearly care about 
what is provided to learners and have worked hard to make sure learners get it all—top quality instructors 
with professional knowledge and the best, most current skills. Being part of this group drives me much 
more than any paycheck ever could.” 



123456789 

Years with PBS TeacherLine 


Figure 3: Number of years instructors have taught for PBS TeacherLine 
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4. High-quality instruction contributed to favorable learner re-enrollment rates during the 
study. 

PBS TeacherLine estimates their learner re-enrollment rate for the five years including the fall 2009, 
winter 2010 and spring 2010 course terms at 56%. During the study, 96.3 percent of learners revealed the 
primary reason why they have taken or would take another course from PBS TeacherLine is the high 
quality of their instructors. One learner wrote 

[The instructor] is, by far, the best instructor 1 have had to date in any online course 1 have taken. 

I would take a course with her again and will recommend her and PBS TeacherLine to my 
colleagues. 

Most learners said they would enroll in another TeacherLine course and recommend TeacherLine courses 
to their colleagues as a direct result of their instructor’s high quality, with statistically insignificant 
differences across course terms. 

5. Favorable learner course grades correlated with instructor quality as measured during 
this study 

Learners’ course grades did not fluctuate systematically by course term, instructor demographic variables, 
or an instructor’s stage in the PD process during this study. However, learner grades did fluctuate by 
instructor quality as measured by the program rubric (Figure 4.) 


35 

o 



A n=3386 (80.7%) B n=373 (8.9%) C n=86 (2.0%) F n=352(8.4%) 

Learner Course Grade (n=4197) 

Figure 4: Learner course grades and their instructors’ quality rating 

VI. DISCUSSION 

One of the key findings from this study supports the conclusions of many others - online instructor 
quality matters. This underscores the importance of providing high quality PD for supporting online 
instructors. Results from this study showed that PBS TeacherLine’s PD model implements various 
professional guidelines established by SREB, iNACOL and current research such as: 

• Modeling high quality online instruction by positioning online teachers as learners in the online 
environment. 

• Offering a multifaceted PD model that includes online courses, mentoring, static materials and an 
online professional learning community. 

• Clearly stating and demonstrating expectations for high quality instruction originating with 
research-based best practices. 

• Leveraging technology to accurately measure PD quality and instructional performance from 
multiple perspectives by aggregating data for easy analysis and discussion. 
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• Utilizing the online environment to provide accurate, timely and iterative feedback about 
instructional quality to instructors, prompts for responding to the feedback, and an asynchronous 
forum for sharing ideas that can improve online instruction. 

• Focusing PD content on: 

o Research-tested strategies that highlight effective ways of teaching and learning online, 
o Extensive modeling of new techniques and guided practice to scaffold learning, 
o Embedded and problem-based learning opportunities with immediate applications to real- 
world environments conveyed online. 

Implementing these strategies has contributed positively to instructor satisfaction, high quality online 
instruction, and learner outcomes such as academic success and favorable re-enrollment statistics. While 
post-hoc assessments of the quality of PBS TeacherLine’s PD offerings and instructor competencies 
cannot establish a causal relationship between the two, both instructors and course learners are satisfied 
and successful. 

This study also explores measures that can provide useful starting points for other PD programs looking 
to implement a continuous improvement cycle. For example, formative assessment such as 3D reviews 
can be effective for achieving various PD outcomes by shaping high quality online instruction, as rated by 
program-level staff, instructors, and learners. Going forward, PBS TeacherLine will provide new 
instructors with two course reviews during their first year and one review during subsequent calendar 
years in order to meet their continuous improvement objectives. Scales for all measures will become 
more sensitive than the three-point scale piloted during the study. The course review document itself also 
serves as a record of performance and PD completion that can be used until credentialed PD opportunities 
for online faculty become more widespread. Providing such a document to online faculty responds to 
their need for artifacts that capture instructional strengths and PD priorities, and also align with 
contemporary discussions about online teaching and learning. 

More research is needed to determine whether and how specific PD elements are effective in other 
environments. Additionally, it will be important to determine which PD elements implemented here, if 
any, may not be necessary in order to achieve favorable outcomes. Comparison testing with validated 
measures and benchmarking will significantly improve our understanding in these areas. A more robust 
analysis across sectors (higher education, k-12, industry) and programs involving well-constructed 
measures will improve the generalizability of these findings. 

The implications for this study’s findings are limited somewhat by factors related to its design and 
sample. The study did not utilize a comparison group or randomization of participants into a group that 
received PBS TeacherLine’s PD. This allowed for careful study of the relationship between PD 
implementation and various outcomes for all instructors, but doesn’t provide data for contextualizing 
gains relative to other PD conditions. 


VII. CONCLUSION 

Broadly, this study is an example of a systematic attempt to better understand how PD for online 
instructors that are using professional standards from organizations such as iNACOL and SREB may 
contribute to positive outcomes that are relevant to the online learning environment. Within this study, 
data suggest implementing PD requirements may contribute to learner success, particularly among adult 
learners. In most instances, instructors who performed well on their course reviews also received very 
positive feedback from learners. A logical next step is to further explore and test this relationship in 
different settings to enhance the discussion of learner outcomes associated with high-quality PD for 
online course instructors. 

Others are encouraged to consider PBS TeacherLine’s PD model as a guide and customize the application 
of broader concepts to address the particular information priorities that will best support their goals for 
data-driven practice. For example, the PD experience may be more relevant with built-in weekly, daily or 
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even real-time instructional data such as those available through data visualization and dashboards for 
rapid processing. Continued review and discussion of empirical data on PD for supporting online 
instructors will further illustrate best practices suggested by current professional guidelines. 
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X. APPENDIX A: 3D COURSE REVIEW 
LEVEL I 


Appendices may be downloaded from altasolutionsgroup.com 

This information provides feedback on your facilitation from three perspectives: your self-evaluation , the 
PBS Associate Director of Online Facilitation, and aggregate survey results from learners in your 
course(s). 

PBS TeacherLine 
Facilitator Course Review 
Facilitator Name: 

Section Number: 
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Course Term: 

Number of Learners: 

5 = Outstanding 4 = Excellent 3 = Very good 2 = Satisfactory 1 = Needs improvement 


Course climate and community building 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Providing, modeling and/or enforcing 
appropriate standards for learner behavior. 









Creating a safe and comfortable online 
learning environment. 









Utilizing effective techniques for 
communicating in a friendly way and 
remaining accessible to learners. 









PBS Comments: 


Instruction 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Strengthening learners’ understanding of 
course concepts through various interactions 
(e.g. discussion boards, assessment feedback). 









Utilizing techniques for explaining concepts 
clearly and effectively. 









PBS Comments: 

Interaction and discussion facilitation 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Encouraging a range of viewpoints in 
discussions. 









Encouraging learners’ continued 
interaction/engagement through the use of 
questions or comments. 









Synthesizing learner posts and stimulating 
ongoing discussion on the discussion boards. 









Demonstrating high-quality written 
communication skills throughout the course. 









PBS Comments: 


Course organization 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 
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Coordinating and assisting learners in 
understanding course requirements and 
procedures for working online. 









Personalizing the course to accommodate a 
range of learning styles. 









Responding to learner questions within 24 
hours. 









Remaining sensitive to any difficulties that 
learners have completing assignments or 
maintaining the course pace. 









PBS Comments: 


Implementation of assessments 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Keeping learners informed of their progress 
during the course. 









Providing detailed, thoughtful and 
constructive feedback about learners’ 
performance in the course. 



* 

* 

* 

* 

* 

* 

Providing detailed and thoughtful feedback 
about learners’ performance in the course. 

* 

* 







Providing constructive feedback about 
learners’ performance in the course. 

* 

* 







PBS Comments: 


*Question was combined on reflection log and PBS TL course review rubric, and split between two 
questions on the learner survey 


Other 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Conducting the course in accordance with 
facilitator checklist and contract deliverables. 



* 

* 

* 

* 

* 

* 

PBS Comments: 


* This question was not asked of learners 


Strengths that your learners associated with you: 

Suggestions from your learners for how this course could have been facilitated more effectively: 
Self Reflection Comments 

How you used the information contained in your course review: 

Your strengths as a TeacherLine facilitator the data in your course review suggest: 

Areas for improvement the data in your course review suggest: 

What you found most striking about the data in your course review: 

Your goals that address weaknesses suggested by the data in your course review: 

How you will carry out these goals: 
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LEVEL II 

This information provides feedback on your facilitation from three perspectives: your self-evaluation , the 
PBS Associate Director of Online Facilitation , and aggregate sur\’ey results from learners in your 

course(s). 

PBS TeacherLine 
Facilitator Course Review 
Facilitator Name: 

Section Number: 

Course Term: 

Number of Learners: 

5 = Outstanding 4 = Excellent 3 = Very good 2 = Satisfactory l = Needs improvement 


Course climate and community building 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Providing, modeling and/or enforcing 
appropriate standards for learner behavior. 









Creating a safe and comfortable online 
learning environment. 









Utilizing effective techniques for 
communicating in a friendly way and 
remaining accessible to learners. 









PBS Comments: 


Instruction 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Strengthening learners’ understanding of 
course concepts through various interactions 
(e.g. discussion boards, assessment 
feedback). 









Utilizing techniques for explaining concepts 
clearly and effectively. 









PBS Comments: 

Interaction and discussion facilitation 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Encouraging a range of viewpoints in 
discussions. 









Encouraging learners’ continued 
interaction/engagement through the use of 
questions or comments. 









Synthesizing learner posts and stimulating 
ongoing discussion on the discussion boards. 









Demonstrating high-quality written 
communication skills throughout the course. 









PBS Comments: 
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Course organization 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Coordinating and assisting learners in 
understanding course requirements and 
procedures for working online. 









Personalizing the course to accommodate a 
range of learning styles. 









Responding to learner questions within 24 
hours. 









Remaining sensitive to any difficulties that 
learners have completing assignments or 
maintaining the course pace. 









PBS Comments: 


Implementation of assessments 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Keeping learners informed of their progress 
during the course. 









Providing detailed, thoughtful and 
constructive feedback about learners’ 
performance in the course. 



* 

* 

* 

* 

* 

* 

Providing detailed and thoughtful feedback 
about learners’ performance in the course. 

* 

* 







Providing constructive feedback about 
learners’ performance in the course. 

* 

* 







PBS Comments: 


*Question was combined on self reflection log and PBS course review rubrics and split between two 
questions on learner survey 


Other 

Self 

PBS 

Learners 

Mean 

5 

4 

3 

2 

1 

Conducting the course in accordance with 
facilitator checklist and contract deliverables. 



* 

* 

* 

* 

* 

* 

PBS Comments: 


* This question was not asked of learners 

Self Reflection Comments 

How you used the information contained in your course review: 

Your strengths as a TeacherLine facilitator the data in your course review suggest: 

Areas for improvement the data in your course review suggest: 

What you found most striking about the data in your course review: 

Your goals that address weaknesses suggested by the data in your course review: 

How you will carry out these goals: 

The following tables compare feedback on the facilitation of your previous course(s) to the feedback 
presented above for your Summer 2010 course(s). The data represents an aggregate of responses from 
all courses facilitated in each term and is intended to compare feedback on your facilitation in general, 
rather than on a particular course you may have facilitated more than once. 
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Previous-Summer 2010 
Course Review Comparison 
Fall 2009 Course(s): 

Winter 2010 Course(s): 

Spring 2010 Course(s): 

Summer 2010 Course(s): 

F 09, W10, and Sp 10: 3 = Excellent 2 = Satisfactory> 1 = Needs Improvement 

S 10: 5 = Outstanding 4 = Excellent 3 = Very good 2 = Satisfactory 1 = Needs 

improvement 


Course climate and 
community building 

Self 

PBS 

Learners 

F 09 

W10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

s 

10 

Providing, modeling 
and/or enforcing 
appropriate standards 
for learner behavior. 













Creating a safe and 
comfortable online 
learning environment. 













Utilizing effective 
techniques for 
communicating in a 
friendly way and 
remaining accessible to 
learners. 














Instruction 

Self 

PBS 

Learners 

F 09 

W10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

S 10 

F 09 

W 

10 

Sp 

10 

s 

10 

Strengthening 

learners’ 

understanding of 
course concepts 
through various 
interactions (e.g. 
discussion boards, 
assessment feedback). 













Utilizing techniques 
for explaining 
concepts clearly and 
effectively. 
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Interaction and Discussion 
Facilitation 

Self 

PBS 

Learners 

F 09 

W 

10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

s 

10 

Encouraging a range of 
viewpoints in discussions. 













Encouraging learners’ continued 
interaction/engagement through 
the use of questions or 
comments. 













Synthesizing learner posts and 
stimulating ongoing discussion 
on the discussion boards. 













Demonstrating high-quality 
written communication skills 
throughout the course. 














Course 

Organization 

Self 

PBS 

Learners 

F 09 

W10 

Sp 

10 

S 10 

F 09 

W 

10 

Sp 

10 

S 10 

F 09 

W 

10 

Sp 

10 

s 

10 

Coordinating and 
assisting learners in 
understanding course 
requirements and 
procedures for 
working online. 













Personalizing the 
course to 

accommodate a range 
of learning styles. 













Responding to 
learner questions 
within 24 hours. 













Remaining sensitive 
to any difficulties 
that learners have 
completing 
assignments or 
maintaining the 
course pace. 














Implementation of 
Assessments 

Self 

PBS 

Learners 

F 09 

W 

10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

s 

10 

Keeping learners 
informed of their 
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progress during the 
course. 













Providing detailed, 
thoughtful and 
constructive feedback 
about learners’ 
performance in the 
course. 









* 

* 

* 

* 

Providing detailed and 
thoughtful feedback 
about learners’ 
performance in the 
course. 




* 




* 





Providing constructive 
feedback about learners’ 
performance in the 
course. 




* 




* 






*Question was combined on self and PBS survey’s and split between two questions on learner survey 


Other 

Self 

PBS 

Learners 

F 09 

W 

10 

Sp 

10 

S10 

F 09 

W 

10 

Sp 

10 

S 10 

F 09 

W 

10 

Sp 

10 

S 10 

Conducting the 
course in accordance 
with facilitator 
checklist and contract 
deliverables. 









* 

* 

* 

* 


* This question was not asked of learners 


XL APPENDIX B: INSTRUCTOR BACKGROUND SURVEY 

1. Which of the following best describes the professional development that you have completed through 
TeacherLine? 

Online Facilitator Training I (TECH 525/526) 

Online Facilitator Training I (TECH 525/526) plus mentoring 

Online Facilitator Training I (TECH 525/526) plus Online Facilitator Training II (TECH 531) 

Other (please specify) 

2. Overall, to what extent has this professional development prepared you to facilitate 

a TeacherLine course? Not at all, Somewhat, Extensively 

3. Have you taken any professional development for online facilitation, offered by someone other than 
TeacherLine, that has significantly influenced how you facilitate TeacherLine courses? Yes, No 

4. Please describe the primary content, format and duration of any professional development for online 
facilitation, offered by someone other than TeacherLine, that has significantly influenced how you 
facilitate TeacherLine courses (3 sentences max): 

4a. Total number of hours completed: 

4b. In what way(s) has/have this non-TeacherLine professional development influenced your facilitation 
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of TeacherLine courses? 

5. Please use the following scale to rate the quality of the preparation you have received from PBS 
TeacherLine on how to facilitate online courses to date. 

3 - TeacherLine training prepared me to implement this immediately 

2 - TeacherLine training gave me good ideas on how to start, but 1 am not ready to implement 
1 - The training covered this topic but 1 don’t feel at all prepared to apply it in my teaching 
0 - I have not received any training on this 
5a. Course Climate and Community Building 


Providing, modeling and/or enforcing appropriate 
standards for learner behavior. 

3 

2 

1 

0 

Ensuring that learners knew one another and that 
each learner feels comfortable interacting with 
other learners online. 

3 

2 

1 

0 

Techniques for communicating in a friendly way 
and being accessible to learners. 

3 

2 

1 

0 


5b. Instruction 


Stressing important points in information 
resources (e.g. discussion threads, readings) 

3 

2 

1 

0 

Displaying my knowledge of the subject matter. 

3 

2 

1 

0 

Strengthening learners’ understanding of course 
concepts through various interactions (e.g. 
discussion boards, assessment feedback). 

3 

2 

1 

0 

Techniques for explaining concepts clearly and 
effectively. 

3 

2 

1 

0 


5c. Interaction and discussion facilitation 


Encouraging a range of viewpoints in discussions. 

3 

2 

1 

0 

Encouraging my continued interaction/engagement 
through the use of questions or comments. 

3 

2 

1 

0 

Synthesizing learner posts and stimulating ongoing 
discussion on the discussion boards. 

3 

2 

1 

0 

Demonstrating high-quality written communication 
skills throughout the course. 

3 

2 

1 

0 


5d. Course Organization 


Coordinating and assisting learners in 

understanding course requirements and procedures 
for working online. 

3 

2 

1 

0 

Personalizing the course to accommodate a range 
of learning styles. 

3 

2 

1 

0 

Responding to learner questions within 24 hours. 

3 

2 

1 

0 

Remaining sensitive to any difficulties that 
learners have completing assignments or 
maintaining the course pace. 

3 

2 

1 

0 


5e. Implementation of Assessments 
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Keeping learners informed of their progress during 
the course. 

3 

2 

1 

0 

Providing detailed, thoughtful, and constructive 
feedback about learners’ performance in the 
course. 

3 

2 

1 

0 


6. Please rate the professional development content that you need most in order to deepen your online 
facilitation. 

Very high need, High need, Neither high nor low need, Low need, Very low need 
6a. Subject-area content (e.g. Algebra, fictional literature) 

6b. LMS and/or technology content such as web-based social networking tools, webinar 
software, web browsers and file management 

6c. Self-assessing content knowledge and/or pedagogical skills necessary to be an effective 
online facilitator 

6d. Self-assessing abilities to use essential hardware and software to facilitate effectively 
6e. Understanding the instructional design of TeacherLine courses 

6f. Understanding and teaching 21st Century Literacy Skills including: strong academic and 
teamwork skills, thinking and reasoning skills, and proficiency in using technology 

6g. Actively engaging learners by modeling effective communication strategies (e.g. 
asking better questions, being more receptive to feedback) 

6h. Locating and/or critiquing supplemental materials that provide learners with multiple 

opportunities for interacting with TeacherLine course content 

6i. Instructional practices that build on knowledge that learners already have 

6j. Multicultural approaches that reflect the cultural backgrounds of learners 

6k. Strategies that help ESL learners (or help learners work with their ESL students) 

61. Techniques that promote student engagement (e.g. implementing deadlines within the 
course, encouraging active participation, etc.) 

6m. Knowing when and how to evaluate the learning of the individual and the class, 
including acknowledging progress and celebrating achievement 
6n. Using, advocating and modeling safe, legal and ethical use of digital information 
6o. Other (please list) 

7. Please rate the format(s) that you would be most likely to attend for acquiring the above professional 
development content. 

Very likely to attend, Likely to attend, Unlikely to attend, Very unlikely to attend 

7a. Viewing models of effective online teaching, communications and strategies 

7b. Role-playing situations that apply content 

7c. Problem-based discussions with other TeacherLine facilitators 

7d. Problem-based discussions with non- TeacherLine facilitators 

7e. Coaching/mentoring 

7f. Reviewing current research and/or best practices 

7g. Hands-on training and experimentation with technology tools ad applications used to 
deliver instruction 
7h. Other (please list) 

8. What would be necessary to make you want to immediately implement strategies/techniques you have 
learned during professional development (3 sentences max)? 
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9. To what extent do you feel you have received: Inadequate Adequate Exemplary 

9a. Feedback on your performance as a TeacherLine course facilitator 

9b. Opportunities through TeacherLine to deepen the knowledge necessary to be a more 
effective online facilitator 

9c. Opportunities through TeacherLine to deepen the skills necessary to be a more 
effective online facilitator 

9d. Opportunities though a service other than TeacherLine to deepen the knowledge 
necessary to be a more effective online facilitator 

9e. Opportunities though a service other than TeacherLine to deepen the skills necessary 
to be a more effective online facilitator 

9f. Access to technical support that reliably solves technology problems 

10. Briefly describe the most positive aspect of the support you have received as a TeacherLine course 
facilitator to date. This can include professional development, technical support or something else. 


11. Briefly describe the aspect of the support you have received as a TeacherLine course facilitator to date 
that could be most improved. This can include professional development, technical support or something 
else. 

12. Please select the statement below that most closely describes the overall contribution of the FUN to 
your facilitation experience: 

The FUN is essential for successful TeacherLine course facilitation. 

The FUN enhanced my skills and/or abilities, but in a way that 1 could have gotten elsewhere. 

The FUN enhanced my skills and/or abilities in a truly unique way. 

The FUN is a nice professional networking tool, but doesn’t otherwise contribute much to my 
facilitation experience. 

1 refer to the FUN to gain information and/or networking more than 1 contribute content to the 

site. 

1 have never used the FUN 


Generally, the FUN has not impacted my facilitation experience positively or negatively. 
Instructor Reflection Logs (Lv 1 and 11) 


Please use the following scale to self assess the course you just finished teaching. 

5 - Outstanding 4 - Excellent 3 - Very Good 2 - Satisfactory 1 - Needs Improvement 


1. Course Climate and Community Building 


2 . 


1 provided, modeled and/or enforced appropriate 
standards for learner behavior. 

1 

2 

3 

4 

5 

1 created a safe and comfortable online learning 
environment. 

1 

2 

3 

4 

5 

1 utilized effective techniques for communicating 
in a friendly way and remaining accessible to 
learners. 

1 

2 

3 

4 

5 


Instruction 


1 strengthened learners’ understanding of course 
concepts through various interactions (e.g. 
discussion boards, assessment, feedback). 

1 

2 

3 

4 

5 

1 utilized techniques for explaining concepts 

1 

2 

3 

4 

5 
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clearly and effectively. 







3. Interaction and discussion facilitation 


1 encouraged a range of viewpoints in discussions. 

1 

2 

3 

4 

5 

1 encouraged learners’ continued 

interaction/engagement through the use of 
questions or comments. 

1 

2 

3 

4 

5 

1 synthesized learner posts and stimulating ongoing 
discussion on the discussion boards. 

1 

2 

3 

4 

5 

1 demonstrated high-quality written 

communication skills throughout the course. 

1 

2 

3 

4 

5 


4. Course Organization 


1 coordinated and assisted learners in 
understanding course requirements and procedures 
for working online. 

1 

2 

3 

4 

5 

1 personalized the course to accommodate a range 
of learning styles. 

1 

2 

3 

4 

5 

1 responded to learner questions within 24 hours. 

1 

2 

3 

4 

5 

1 remained sensitive to any difficulties that 
learners have completing assignments or 
maintaining the course pace. 

1 

2 

3 

4 

5 


5. Implementation of Assessments 


1 kept learners informed of their progress during 
the course. 

1 

2 

3 

4 

5 

1 provided detailed, thoughtful, and constructive 
feedback about learners’ performance in the 
course. 

1 

2 

3 

4 

5 


6. Other 


1 conducted the course in accordance with 
facilitator checklist and contract deliverables. 

1 

2 

3 

4 

5 


7. Do you have suggestions about how to improve the content or presentation of the core course 
curriculum (provided by TeacherLine developers)? 

8. Briefly describe one or two aspects of your instruction this term (e.g. interaction with students, 
facilitation of the discussion board, supplemental content additions, etc.) that you feel are reflective of 
your strengths as a TeacherLine facilitator. 

9. Create a priority list of your plans for improving your facilitation skills and encouraging your 
professional growth as a TeacherLine facilitator. How will you carry out those plans? 

The following questions were added to differentiate Level 11 from Level 1 logs: 

Last term, you received a course review that incorporated data from PBS TeacherLine, your learners and 
your personal reflection. Please answer the following questions based on the personalized course review 
document that you received. 

10. When did you open the course review? 

Within 24 hours of receiving it 

Within 2-3 days of receiving it 
Within 1 week of receiving it 
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More than 1 week after receiving it 
I have not opened my personalized course review 

11. How many times have you opened the course review document? 

Never Once Twice Three times Four times More than four times 

12. How much time have you spent, in total, reviewing this document? 

0-30 minutes 

31 -60 minutes 
1-1.5 hours 
1.5-2 hours 
Longer than 2 hours 

13. How useful was the information contained in the course review? 

14. Please describe how you have used the information contained in the course review. 

15. What strengths do the data in the course review suggest? 

16. What areas of improvement do the data in the course review suggest? 

17. What was most striking to you about the data in the course review? 

18. After reflecting on the information contained in your personalized course review, please set at least 
two goals that address weaknesses suggested by the data. 

18a. How will you carry out these goals? 

19. Please describe how you would like PBS TeacherLine to use the information in the course review. 

20. What did you like best about the course review process? 

21. How could the course review process be improved? 

XII. APPENDIX C: COURSE REFLECTION LOG 

Instructor Reflection Logs (Lv 1 and 11) 


Please use the following scale to self assess the course you just finished teaching. 

5 - Outstanding 4-Excellent 3-Very Good 2 - Satisfactory 1 - Needs Improvement 

1. Course Climate and Community Building 


1 provided, modeled and/or enforced appropriate 
standards for learner behavior. 

1 

2 

3 

4 

5 

1 created a safe and comfortable online learning 
environment. 

1 

2 

3 

4 

5 

I utilized effective techniques for communicating 
in a friendly way and remaining accessible to 
learners. 

1 

2 

3 

4 

5 


2. Instruction 


I strengthened learners’ understanding of course 
concepts through various interactions (e.g. 
discussion boards, assessment, feedback). 

1 

2 

3 

4 

5 

1 utilized techniques for explaining concepts 
clearly and effectively. 

1 

2 

3 

4 

5 


3. Interaction and discussion facilitation 


I encouraged a range of viewpoints in discussions. 

1 

2 

3 

4 

5 

I encouraged learners’ continued 

1 

2 

3 

4 

5 
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interaction/engagement through the use of 
questions or comments. 






1 synthesized learner posts and stimulating ongoing 
discussion on the discussion boards. 

1 

2 

3 

4 

5 

1 demonstrated high-quality written 
communication skills throughout the course. 

1 

2 

3 

4 

5 


4. Course Organization 


1 coordinated and assisted learners in 
understanding course requirements and procedures 
for working online. 

1 

2 

3 

4 

5 

1 personalized the course to accommodate a range 
of learning styles. 

1 

2 

3 

4 

5 

1 responded to learner questions within 24 hours. 

1 

2 

3 

4 

5 

1 remained sensitive to any difficulties that 
learners have completing assignments or 
maintaining the course pace. 

1 

2 

3 

4 

5 


5. Implementation of Assessments 


1 kept learners informed of their progress during 
the course. 

1 

2 

3 

4 

5 

1 provided detailed, thoughtful, and constructive 
feedback about learners’ performance in the 
course. 

1 

2 

3 

4 

5 

6. Other 

1 conducted the course in accordance with 
facilitator checklist and contract deliverables. 

1 

2 

3 

4 

5 


7. Do you have suggestions about how to improve the content or presentation of the core course 
curriculum (provided by TeacherLine developers)? 

8. Briefly describe one or two aspects of your instruction this term (e.g. interaction with students, 
facilitation of the discussion board, supplemental content additions, etc.) that you feel are reflective of 
your strengths as a TeacherLine facilitator. 

9. Create a priority list of your plans for improving your facilitation skills and encouraging your 
professional growth as a TeacherLine facilitator. How will you carry out those plans? 

The following questions were added to differentiate Level II from Level I logs: 

Last term, you received a course review that incorporated data from PBS TeacherLine, your learners and 
your personal reflection. Please answer the following questions based on the personalized course review 
document that you received. 

10. When did you open the course review? 

Within 24 hours of receiving it 
Within 2-3 days of receiving it 
Within 1 week of receiving it 

More than 1 week after receiving it 
I have not opened my personalized course review 

11. How many times have you opened the course review document? 

Never 

Once 

Twice 
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Three times 

Four times 

More than four times 

12. How much time have you spent, in total, reviewing this document? 

0-30 minutes 

31-60 minutes 
1-1.5 hours 
1.5-2 hours 
Longer than 2 hours 

13. How useful was the information contained in the course review? 

14. Please describe how you have used the information contained in the course review. 

15. What strengths do the data in the course review suggest? 

16. What areas of improvement do the data in the course review suggest? 

17. What was most striking to you about the data in the course review? 

18. After reflecting on the information contained in your personalized course review, please set at least 

two goals that address weaknesses suggested by the data. 

18a. How will you carry out these goals? 

19. Please describe how you would like PBS TeacherLine to use the information in the course review. 

20. What did you like best about the course review process? 

21. How could the course review process be improved? 

XIII. APPENDIX D: PBS TEACHERLINE COURSE REVIEW RUBRIC 

Post-Course Learner Survey 

Please indicate the TeacherLine course code for the course that you just completed (also provided to you 
by your facilitator): 

Facilitator name: 

Course term: 

1. Did you just complete your first TeacherLine course? 

la. If no, how many TeacherLine courses have you taken to date? 

2. How many TeacherLine courses have you taken from this facilitator (including this course) 

3. Please use the following scale to rate the facilitator’s performance during the course that you just 

completed: 

5 - Outstanding 4 - Excellent 3 - Very Good 2 - Satisfactory 1 - Needs Improvement 
3a. Course Climate and Community Building 


The facilitator provided, modeled and/or enforced 
appropriate standards for learner behavior. 

1 

2 

3 

4 

5 

The facilitator created a safe and comfortable 
online learning environment. 

1 

2 

3 

4 

5 

The facilitator utilized effective techniques for 
communicating in a friendly way and remaining 
accessible to learners. 

1 

2 

3 

4 

5 


3b. Instruction 


The facilitator strengthened learners’ 
understanding of course concepts through various 
interactions (e.g. discussion boards, assessment, 
feedback). 

1 

2 

3 

4 

5 

The facilitator utilized techniques for explaining 
concepts clearly and effectively. 

1 

2 

3 

4 

5 
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3c. Interaction and discussion facilitation 


The facilitator encouraged a range of viewpoints in 
discussions. 

1 

2 

3 

4 

5 

The facilitator encouraged learners’ continued 
interaction/engagement through the use of 
questions or comments. 

1 

2 

3 

4 

5 

The facilitator synthesized learner posts and 
stimulating ongoing discussion on the discussion 
boards. 

1 

2 

3 

4 

5 

The facilitator demonstrated high-quality written 
communication skills throughout the course. 

1 

2 

3 

4 

5 


3d. Course Organization 


The facilitator coordinated and assisted learners in 
understanding course requirements and procedures 
for working online. 

1 

2 

3 

4 

5 

The facilitator personalized the course to 
accommodate a range of learning styles. 

1 

2 

3 

4 

5 

The facilitator responded to learner questions 
within 24 hours. 

1 

2 

3 

4 

5 

The facilitator remained sensitive to any 
difficulties that learners have completing 
assignments or maintaining the course pace. 

1 

2 

3 

4 

5 


3e. Implementation of Assessments 


The facilitator kept learners informed of their 
progress during the course. 

1 

2 

3 

4 

5 

The facilitator provided detailed, thoughtful, and 
constructive feedback about learners’ performance 
in the course. 

1 

2 

3 

4 

5 


4. My grade in this course accurately reflects my mastery of course objectives. 

Yes, no 

5. Because of this course facilitator’s knowledge and skills, 1 would/will: 

Definitely not, I doubt it, I might consider it, I am pretty sure I would, Absolutely 

a. Take another TeacherLine course 

b. Take another TeacherLine course from this facilitator 

c. Be able to apply what 1 have learned immediately in my professional practice 

d. Stay connected with other learners beyond this course Recommend this course to a colleague or 
friend 

e. Recommend a TeacherLine course to a colleague or friend 

f. Recommend a TeacherLine course taught by this facilitator to a colleague or friend 

g. Seek training on how to be a TeacherLine course facilitator Other 

h. Other (please indicate here) 

6. What do you consider to be this facilitator's strengths? 

7. Please evaluate, given what you know and have experienced, the performance of this facilitator in 
relation to this specific course. Do you have suggestions for how this course may have been facilitated 
more effectively? 

8. Do you have any suggestions for how to improve the content or presentation of this course's content? 
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XIV. APPENDIX E: POST-COURSE LEARNER SURVEY 

Post-Course Learner Survey 

Please indicate the TeacherLine course code for the course that you just completed (also provided to you 
by your facilitator): 

Facilitator name: 

Course term: 

1. Did you just complete your first TeacherLine course? 

la. If no, how many TeacherLine courses have you taken to date? 

2. How many TeacherLine courses have you taken from this facilitator (including this course) 

3. Please use the following scale to rate the facilitator’s performance during the course that you just 
completed: 


5 - Outstanding 4 - Excellent 3 - Very Good 2 - Satisfactory 1 - Needs Improvement 
3a. Course Climate and Community Building 


The facilitator provided, modeled and/or enforced 
appropriate standards for learner behavior. 

1 

2 

3 

4 

5 

The facilitator created a safe and comfortable 
online learning environment. 

1 

2 

3 

4 

5 

The facilitator utilized effective techniques for 
communicating in a friendly way and remaining 
accessible to learners. 

1 

2 

3 

4 

5 


3b. Instruction 


The facilitator strengthened learners’ 
understanding of course concepts through various 
interactions (e.g. discussion boards, assessment, 
feedback). 

1 

2 

3 

4 

5 

The facilitator utilized techniques for explaining 
concepts clearly and effectively. 

1 

2 

3 

4 

5 


3 c. Interaction and discussion facilitation 


The facilitator encouraged a range of viewpoints in 
discussions. 

1 

2 

3 

4 

5 

The facilitator encouraged learners’ continued 
interaction/engagement through the use of 
questions or comments. 

1 

2 

3 

4 

5 

The facilitator synthesized learner posts and 
stimulating ongoing discussion on the discussion 
boards. 

1 

2 

3 

4 

5 

The facilitator demonstrated high-quality written 
communication skills throughout the course. 

1 

2 

3 

4 

5 


3d. Course Organization 


The facilitator coordinated and assisted learners in 
understanding course requirements and procedures 
for working online. 

1 

2 

3 

4 

5 

The facilitator personalized the course to 
accommodate a range of learning styles. 

1 

2 

3 

4 

5 

The facilitator responded to learner questions 

1 

2 

3 

4 

5 
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within 24 hours. 






The facilitator remained sensitive to any 
difficulties that learners have completing 
assignments or maintaining the course pace. 

1 

2 

3 

4 

5 


3e. Implementation of Assessments 


The facilitator kept learners informed of their 
progress during the course. 

1 

2 

3 

4 

5 

The facilitator provided detailed, thoughtful, and 
constructive feedback about learners’ performance 
in the course. 

1 

2 

3 

4 

5 


4. My grade in this course accurately reflects my mastery of course objectives. 

Yes, no 

5. Because of this course facilitator’s knowledge and skills, 1 would/will: 

Definitely not, I doubt it, I might consider it, I am pretty sure I would, Absolutely 

a. Take another TeacherLine course 

b. Take another TeacherLine course from this facilitator 

c. Be able to apply what 1 have learned immediately in my professional practice 

d. Stay connected with other learners beyond this course Recommend this course to a colleague or 
friend 

e. Recommend a TeacherLine course to a colleague or friend 

f. Recommend a TeacherLine course taught by this facilitator to a colleague or friend 

g. Seek training on how to be a TeacherLine course facilitator Other 

h. Other (please indicate here) 

6. What do you consider to be this facilitator's strengths? 

7. Please evaluate, given what you know and have experienced, the performance of this facilitator in 
relation to this specific course. Do you have suggestions for how this course may have been facilitated 
more effectively? 

8. Do you have any suggestions for how to improve the content or presentation of this course's content? 

XV. APPENDIX F: COURSE FACILITATOR INTERVIEWS 

Participants: Lv 1 course facilitators (W2010) 

Duration: 15-20 minutes each 
Questions: 

• What do you feel are your strongest facilitation skills? 

• What are the facilitation skills that you feel you need to most improve? 

• What are your greatest professional development needs in order to be a better online course 
facilitator? 

• How well do you feel that PBS TeacherLine’s current professional development and support 
meets these needs? How could PBS TeacherLine do better? 

• What did you like/not like about completing the reflection log that was part of this research 
study? 

Participants: Lv 11 course facilitators (W2010) 

Duration: 15-20 minutes each 
Questions: 
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• In general, how would you say your facilitation this term compared to last term? What was 
harder/easier about this term? What went better/worse? 

• Did you open the course review that you received after the conclusion of the fall 2009 term? If 
so, please describe how you used the information contained in the course review. What was most 
striking to you about the course review process? If not, why? 

• How would you like PBS TeacherLine to use the information in the course reviews? 

• How could the course review process be improved? 

• Is there anything else about PBS TeacherLine’s support of your facilitation or the research 
process that you would like to share? 


5-point rating 
scale 


XVI. APPENDIX G: Program Rubric 

Points 

0 1 2 


Qualifications (11 points) 

Facilitators must 
have a master’s 
degree in 
education, or a 
Degree/ related field of 
certifications study, as well as 
the appropriate 
subject matter 
certification and 
expertise 


OFT 


Sucessfully 
completed OFT I 


Enrolled in up to 5 
fully online course 
Prior online as a learner or up 
learning to 5 hybrid online 
courses as a 
learner 


Prior online 
teaching 


Independently 
delivered at least 
one fully online 
course for any 
provider 


Facilitator holds a 
CAS in education 
or a related field of 
study including the 
subj ect matter 
topic (s)he 
regularly teaches 
online 

Successfully 
completed OFT I 
and mentoring 

Successfully 
completed up to 5 
fully online non- 
TL courses as a 
learner OR 
successfully 
completed 1-2 PBS 
TL courses as a 
learner 

Independently 
delivered at least 
ten fully online 
courses for any 
non-T eacherLine 
provider OR 
delivered fully 
online instruction 
independently for 
PBS TeacherLine 
for up to 3 course 
terms 


Facilitator holds a 
PhD in education 
or a related field of 
study including the 
subj ect matter 
topic (s)he 
regularly teaches 
online 

Successfully 
completed OFT I 
and II 

Successfully 
completed 6-15 
fully online on-TL 
courses as a 
learner OR 
Successfully 
completed 3-5 
PBS TL courses as 
a leraner 

Independently 
delivered at least 
twenty fully online 
courses for any 
provider OR 
delivered fully 
online instruction 
independently for 
PBS TeacherLine 
for between 3 and 
6 course terms 


3 


Successfully 
completed OFT I 
and II and mentoring 

Successfully 
completed 16 or 
more fully online 
non-TL courses as a 
learner OR 
successfully 
completed 6 or more 
PBS TL courses as a 
learner 

Delivered fully 
online instruction 
independently for 
PBS TeacherLine 
for 7 or more course 
terms OR 
independently 
delivered at least 
one online instructor 
training course for 
PBS TL (i.e. OFT I 
or II) 
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Learner Rating 


Course climate 
and community 
building 


Instruction 


Interaction and 
discussion 
facilitation 


Course 

organization 


Implement¬ 
ation of 
assessments 

PBS TL Rating 
Course climate 
and community 
building 
Instruction 
Interaction 
and discussion 
facilitation 
Course 
organization 
Implement¬ 
ation of 
assessments 
Compliance 
with PBS TL 
content 
checklist and 
required 
deliverables 

Participation 
in course 
review process 

Post-course 
learner surveys 


Performance - evaluated annually (37 points) 


Gathered 
responses from 

< 50% of learners 

Gathered 
responses from 

< 50% of learners 

Gathered 
responses from 

< 50% of learners 

Gathered 
responses from 

< 50% of learners 

Gathered 
responses from 

< 50% of learners 


<2.9 

<2.9 

<2.9 

<2.9 

<2.9 


No 


Incomplete log(s) 


Gathered 
responses from 
50% of learners or 
less 


Aggregate learner 
rating of 3.0 - 3.9 
from < 50% of 
learners 

Aggregate learner 
rating of 3.0 - 3.9 
from < 50% of 
learners 

Aggregate learner 
rating of 3.0 - 3.9 
from < 50% of 
learners 

Aggregate learner 
rating of 3.0 - 3.9 
from < 50% of 
learners 

Aggregate learner 
rating of 3.0 - 3.9 
from < 50% of 
learners 


3.0-3.9 
3.0-3.9 
3.0-3.9 


3.0-3.9 


3.0-3.9 


Yes 


Fully completed 
Log 1 on time 

Achieved 50.1% - 
74.9% response on 
post-course learner 
survey 


Aggregate learner 
rating of 4.0 - 4.4 
from < 50% of 
learners 

Aggregate learner 
rating of 4.0 - 4.4 
from < 50% of 
learners 

Aggregate learner 
rating of 4.0 - 4.4 
from < 50% of 
learners 

Aggregate learner 
rating of 4.0 - 4.4 
from < 50% of 
learners 

Aggregate learner 
rating of 4.0 - 4.4 
from < 50% of 
learners 


4.0-4.4 
4.0-4.4 
4.0-4.4 

4.0-4.4 

4.0-4.4 


Fully completed 
Log 11 on time 

Achieved 75% - 
99.9% response on 
post-course 
learner survey 


Aggregate learner 
rating of 4.5 - 5.0 
from < 50% of 
learners 

Aggregate learner 
rating of 4.5 - 5.0 
from < 50% of 
learners 

Aggregate learner 
rating of 4.5 - 5.0 
from < 50% of 
learners 

Aggregate learner 
rating of 4.5 - 5.0 
from < 50% of 
learners 

Aggregate learner 
rating of 4.5 - 5.0 
from < 50% of 
learners 


4.5-5.0 
4.5-5.0 
4.5-5.0 

4.5-5.0 

4.5-5.0 


Demonstrated 
achievement of 
goals set in Log 11 

Achieved 100% 
response on post¬ 
course learner 
survey 
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Title 

low 

Points 

high 

Facilitator 

0 


16 

Distinguished 

Facilitator 

17 


33 

Master 

Facilitator 

34 


48 


XVII. APPENDIX H: Facilitator Course Review 

Please complete this form for each PBS TeacherLine course facilitator. This form is used to provide 
feedback on the facilitator's performance, and to identify areas of needed professional development. Thi s 
log will help pinpoint their strengths and weaknesses. 

• Facilitator Name: 

• Section Number: 

• Course Term: 

• Reviewer’s Name: 

Please use the following scale to rate the facilitator’s performance during the course that they just 
finished facilitating: 

5 - Outstanding 4 - Excellent 3 - Very Good 2 - Satisfactory 1 - Needs Improvement 
5a. Course Climate and Community Building 
The facilitator provided, modeled and/or enforced 
appropriate standards for learner behavior. 

The facilitator created a safe and comfortable online 
learning environment. 

The facilitator utilized effective techniques for 
communicating in a friendly way and remaining 
accessible to learners. 

5b. Comments: 

6a. Instruction 

The facilitator strengthened learners’ understanding of 
course concepts through various interactions (e.g. 1 2 

discussion boards, assessment, feedback). 

The facilitator utilized techniques for explaining ^ 2 

concepts clearly and effectively. 

6b. Comments: 

7a. Interaction and discussion facilitation 

The facilitator encouraged a range of viewpoints in 12 

discussions. 

The facilitator encouraged learners’ continued 

interaction/engagement through the use of questions or 1 2 3 4 5 

comments. 

The facilitator synthesized learner posts and stimulating 1 2 3 4 5 


3 4 5 

3 4 5 


1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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ongoing discussion on the discussion boards. 

The facilitator demonstrated high-quality written 
communication skills throughout the course. 

7b. Comments: 


8a. Course Organization 

The facilitator coordinated and assisted learners in 
understanding course requirements and procedures for 1 
working online. 

The facilitator personalized the course to accommodate 1 
a range of learning styles. 

The facilitator responded to learner questions within 24 
hours. 

The facilitator remained sensitive to any difficulties 

that learners have completing assignments or 1 

maintaining the course pace. 

8b. Comments: 


2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 


9a. Implementation of Assessments 

The facilitator kept learners informed of their progress 1 2 3 4 5 

during the course. 

The facilitator provided detailed, thoughtful, and 

constructive feedback about learners’ performance in 1 2 3 4 5 

the course. 

9b. Comments: 


10a. Other 

The facilitator conducted the course in accordance with 
facilitator checklist and contract deliverables. 

10b. Comments: 


XVIII: APPENDIX I: Course Facilitation Guidelines 


Facilitation Evaluation 
Guidelines 


PBS TeacherLine* 


This document summarizes the procedures and instruments resulting from the 2009-2010 
formative/summative research study of PBS TeacherLine’s (TL) online course facilitation quality and 
support. Testing and refinement took place during four consecutive course terms, and was guided by 
active input provided by TL leaders, facilitators, and TL course learners. 

PBS TeacherLine utilizes a multi-tiered professional development experience for online course 
facilitators that features activities ranging from online professional development courses to personalized 
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course reviews. The final versions of instruments are designed to gather feedback on each component of 
TeacherLine’s facilitator support and professional development model: 


TL Facilitation Professional Development and Support 


Component 
Evaluating facilitators: 

Personalized course review 


Measure(s) 


Reflection logs 1 and II, TL course rubric, learner 
survey 

Evaluating the TL facilitation program: 


OFT I 
OFT II 
Mentoring 
FUN 

Faculty meetings 
Facilitator qualifications 
Facilitator performance 
Facilitator compliance with 
procedures 

Facilitator participation in 
course review 


Background survey 
Background survey 
Background survey 
Background survey 
Reflection logs I and II 
TL program rubric 
TL program rubric 
TL program rubric 

TL program rubric 


Evaluations of individual facilitators and the entire TL facilitation support program are designed to take 
place annually, and to inform each other. Consistent with TL’s overall commitment to quality and 
applying current research-based best practices, this evaluation procedure calls upon multiple sources to 
assure data integrity. The design balances a need for both individuals and the program to co-evolve and 
improve. It also respects facilitators’ engagement with the process by asking questions that provoke their 
critical reflection while simultaneously collecting data on program and facilitator performance. 

A. Evaluating Facilitators 

Up to two formal evaluations of each facilitator take place annually. Each evaluation uses common 
research-based criteria and scales to rate a facilitator’s performance from three perspectives: facilitator, a 
TL leader, and course learners. PBS TeacherLine aggregates and summarizes these data into a 
personalized course review document that is provided to each facilitator at the end of the term. The 
instruments that facilitators use to reflect on their teaching are differentiated; Lv II logs ask facilitators to 
answer additional questions about a prior course review and have been shown to stimulate critical 
reflection among facilitators. 

Here are the steps involved with evaluating individual TL course facilitators through the course review 
process: 

What to do: 

1. At the start of a course term: 

a. Deploy Lv II reflection logs to all facilitators that have participated in the course review process at 
least once during the prior calendar year; Deploy Lv I reflection logs to everyone else. The goal is to 
have all facilitators complete one Lv II log annually. For newer facilitators, this will require them to 
first complete their Lv I log. 

2. Midway through each course term: 
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a. Deploy the post-course learner survey 

b. Remind facilitators and learners to complete reflection logs and learner surveys respectively 

c. Provide learner survey response rates to each facilitator as the term ends 

d. Conduct TL course review 
3. After each course term ends: 

a. Match each learner’s survey data with his/her facilitator 

b. Calculate response rates for each facilitator 

c. Match facilitator reflection log data with learner data with PBS TL data 

d. Prepare course review and share it with facilitators 


Title 

Points 

Low 

High 

Facilitator 

0 

16 

Distinguished Facilitator 

17 

33 

Master Facilitator 

34 

48 


B. Evaluating TL’s Facilitation Support Program 

PBS TeacherLine’s facilitation support program not only provides personalized performance feedback 
from multiple perspectives, but it embodies many current research-based best practices for professional 
development including peer support, professional learning communities, and problem-based learning. 

PBS TeacherLine’s facilitation support model, compared with research-based best practices 

Research-based best practices PBS TeacherLine 

Peer support Mentoring, FUN 

Ongoing, embedded guidance FUN, Personalized course reviews 

Mentoring, FUN, Assignments within OFT 1 
Problem-based and OFT 11, Faculty meetings, Personalized 

course reviews 

Courses (OFT 1 and 11), Dyadic peer 
Differentiated mentoring, Learning community through the 

FUN, and personalized course reviews 

Online All facets take place online 

Each of PBS TeacherLine’s online course facilitators evaluates the preparation and support that PBS 
TeacherLine has provided to them by completing a background survey. The background survey is 
deployed annually to facilitators who have completed OFT 1 and/or 11 and collects feedback about 
facilitators’ preparation for online instruction, whether provided by PBS TeacherLine or another agency. 
Complementing the background survey is a program rubric that can be used to assess the distribution of 
all facilitators’ performance during a given year. The program can be viewed as a success if at least 60% 
of the facilitators earn 17 out of 48 total points across the following four areas: 

1. Qualifications (11 points) - Recognizes the positive but indirect contribution of continued online 
experience and other professional development accomplishments to demonstrated excellence as a 
TeacherLine facilitator 

a. Degree/Certification 
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b. OFT 1,11 and mentoring 

c. Prior and continued online learning 

d. Prior and continued online teaching 

2. Performance (31 points) -PBS TeacherLine and learners (in aggregate) weigh in on facilitation 
performance using a 5-point Likert scale on the dimensions of: 

a. Course climate and community building 

b. Instruction 

c. Interaction and discussion facilitation 

d. Course organization 

e. Implementation of assessments 

3. Compliance with PBS TL content checklist and required deliverables (1 point) 

4. Participation in course review process including more than 50% response rate on the learner survey (6 
points) 

The rubric can be used to score and differentiate facilitators, and/or to evaluate the effectiveness of TL’s 
facilitation support program. 

Use rubric to score and differentiate facilitators. 

C. Documents: 

Program-level: 

Program rubric 
Background survey 

Facilitator-level: 

Reflection log Lv 1 and Lv 1 3D course review template 
Reflection log Lv 11 and Lv II 3D course review template 
Post-course learner survey 
TL course review rubric 
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